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Developing  Metrics  for  a  Multi-Agency  Harbour  Safety  and  Security  Exercise 


Marine  security,  specifically  port  and  harbour  security,  has  been  identified  as  one  of  the 
Government  of  Canada’s  priorities.  Harbour  security  is  a  complex  problem  because  of  the 
multiplicity  of  threats  -  air,  land  and  water,  and  because  of  the  potential  for  overlapping 
jurisdictions  within  the  harbour  environment.  In  response,  Defence  Research  and  Development 
Canada  -  Atlantic  examined  existing  inter-agency  processes  in  a  major  harbour  and  developed 
metrics  for  use  during  a  live  multi-agency  harbour  safety  and  security  exercise.  This  paper  gives  an 
overview  of  the  metrics  that  were  developed,  and  highlights  the  challenges  encountered  while 
deploying  these  metrics  during  a  live  exercise. 


1 .  Introduction 

Given  present  realities  in  the  security  and  defence  of  North  America,  the  Canadian  government  has 
clearly  articulated  its  commitment  to  improve  marine  security  in  Canada’s  territorial  waters  and  shore 
facilities  (1).  A  key  component  of  this  strategy  is  the  insistence  that  marine  security,  emphasizing  harbour 
security,  be  given  priority  in  the  establishment  of  an  integrated,  interdepartmental  approach.  In  response, 
Defence  Research  and  Development  Canada  (DRDC)  and  Public  Safety  Canada  (PS)  collaborated  on  a 
multi-agency  safety  and  security  exercise  in  Halifax  Harbour.  DRDC  formed  the  Interagency  Harbour 
Security  Coordination  (IHSC)  project  and  PS  designed  and  led  a  large  multi-agency  exercise.  The  IHSC 
project  was  funded  by  the  Marine  Security  Coordination  Fund1,  administered  by  the  Interdepartmental 
Marine  Security  Working  Group  (IMSWG)2.  The  IHSC  focused  on  mapping  interdepartmental 
processes,  such  as  command  and  control  and  information  sharing,  and  developed  metrics  for  use  during  a 
live  multi-agency  harbour  safety  and  security  exercise. 

The  interdepartmental  process  mapping  completed  for  the  IHSC  project  documented  existing  multi¬ 
agency  processes  in  the  Port  of  Halifax.  The  metrics  and  data  collection  focused  on  the  assessment  of 
command  and  control  and  interdepartmental  communications  during  a  live  multi-agency  exercise. 
Exercise  evaluation  and  improvement  planning  guidance  provided  by  the  U.S.  Homeland  Security 
Exercise  and  Evaluation  Program  (HSEEP)  formed  an  integral  part  of  the  early  metrics  development  (2). 
The  live  exercise  was  a  full-scale  marine  safety,  security,  and  consequence  management  exercise  in  the 
Halifax  Harbour  on  October  21,  2009.  It  included  Harbour  stakeholders,  three  levels  of  government  and 
the  private  sector.  The  duration  of  the  full-scale  exercise  (on  site  in  Halifax)  was  10  hours,  with  a 
simulated  national  headquarters  cell  stood  up  in  Ottawa  for  a  period  of  9  hours. 

This  paper  provides  an  overview  of  the  IHSC  project.  The  focus  of  the  discussion  is  on  the  evaluation 
developed  for  the  multi-agency  live  exercise  and  how  it  was  executed.  Effort  has  been  made  to  highlight 
the  challenges  encountered  during  this  project,  and  to  provide  advice  on  how  these  challenges  can 
potentially  be  mitigated. 


1  The  Marine  Security  Coordination  Fund  was  established  to  fund  one-time  or  limited-period  projects  that  intend  to 
enhance  collaboration  and  coordination  of  Federal  departments  and  agencies  on  marine  security  related  activities. 

2  The  Interdepartmental  Marine  Security  Working  group  is  chaired  byTransport  Canada  with  representatives  from 
over  17  Federal  departments  and  agencies. 


2.  Live  Multi-Agency  Safety  and  Security  Exercise:  Scenario  and  Evaluation 
2.1.  Scenario  -  General  Description 

The  general  scenario  for  the  exercise  was  divided  in  two  geographic  areas  with  three  main  events 
triggering  the  response.  The  first  event  involved  a  container  ship  that  developed  a  critical  equipment 
failure  and  was  awaiting  repairs.  While  at  anchor  a  string  of  explosions  on  the  ship  triggered  a  fire  that 
caused  the  subsequent  release  of  bunker  fuel  into  the  Harbour.  After  the  explosion,  the  body  of  a  foreign 
national  non-crewmember  was  found  near  the  stricken  vessel  with  ties  to  a  terrorist  organization.  The 
second  event  involved  the  ignition  of  a  container  of  Sodium  hydrosulphite  causing  a  plume  of  toxic 
smoke  that  was  carried  inland  over  a  densely  populated  urban  area.  The  third  event  involved  a  security 
incident  at  an  oil  production  facility  on  the  Halifax  harbour  that  may  be  connected  to  container  ship 
explosion. 


Figure  1:  Primary  event  location  in  Halifax  Harbour  for  the  simulated  container  ship  fire  and  on-water  emergency 
response. 

2.2.  Scenario  -  Exercise  Assumptions,  Artificialities,  and  Constraints 

Assumptions,  artificiality,  and  constraints  are  a  fact  of  life  during  the  planning  and  execution  of  a  live 
exercise.  The  focus  of  this  section  is  to  highlight  the  factors  that  directly  affected  the  development  of 
metrics  and  the  data  collection  component  of  the  IHSC  project.  Most  of  these  factors  are  not  unique  to  a 
specific  live  exercise,  rather,  they  could  be  considered  systemic  to  multi-level  emergency  management 
exercises. 

2.2.1.  Exercise  Assumptions 

Exercise  assumptions  made  prior  to  developing  the  live  exercise  evaluation  concerned  the  actions  of 
exercise  participants.  The  primary  assumption  was  that  exercise  participants  were  all  well  versed  in  their 
own  department/agency  response  plans  and  procedures,  and  that  participants  would  act  in  accordance 


with  these  existing  plans,  policies,  and  procedures.  In  addition,  it  was  assumed  that  real-world  emergency 
response  actions  would  take  priority  over  exercise  actions. 

2.2.2.  Exercise  Artificialities 

Exercise  artificialities  were  made  to  allow  the  exercise  to  proceed  even  if  it  would  not  proceed  under 
normal  circumstances.  This  included  simulated  support  from  federal  departments’  National  headquarters, 
artificial  weather  and  tidal  information,  use  of  alternative  communication  frequencies,  simulated  maritime 
exclusion  zone  orders,  and  use  of  some  non-standard  equipment.  If  an  expected  action  did  not  occur  as 
described  in  the  Master  Scenario  Events  List  (MSEL)  the  exercise  control  group  would  force  the  action  to 
ensure  the  MSEL  timeline  is  preserved.  Lastly,  some  decision  timelines  were  shortened  to  allow  for 
increased  exercise  play. 

2.2.3.  Constraints 

Several  constraints  were  present  during  the  IHSC  that  influenced  the  development  of  metrics.  Early  in 
the  IHSC,  a  conscious  decision  was  made  to  avoid  evaluating  internal  processes  of  participating  agencies 
and  departments,  instead  focusing  the  evaluation  on  predominately  inter-departmental  processes.  A 
decision  was  also  made  by  the  evaluation  team  to  restrict  video,  voice,  and  digital  recording  at  each  of  the 
operations  centers  due  to  sensitivity  concerns.  Lastly,  the  live  exercise  evaluation  was  designed  to  limit 
the  number  of  interactions  with  exercise  participants  during  exercise  play  to  minimize  interference. 

2.3.  Inter-Agency  Process  Mapping 

The  security  environment  in  Canada’s  ports  continues  to  evolve.  Elements  of  concern  to  policy  makers 
include  threat  diffusion  and  general  expectations  of  the  government’s  ability  to  mitigate  threats  and 
manage  incidents.  When  it  comes  to  consequence  management  the  public  is  not  inclined  to  differentiate 
between  levels  of  government  or  investigate  departmental  mandates.  A  “Whole  of  Government” 
approach  is  both  assumed  and  is  often  necessary.  This  emphasizes  the  need  for  preparation  to  pro¬ 
actively  develop  an  understanding  of  threats,  vulnerabilities,  systemic  capabilities,  shortfalls,  and 
opportunities  to  realize  efficiencies  and  ensure  effects. 

Multi-jurisdictional,  multi-level  (federal,  provincial,  municipal,  and  private  sector)  emergency 
management  exercises  often  show  that  improved  coordination  and  better  interdepartmental  information 
management  protocols  are  needed  among  the  large  number  of  authorities  and  stakeholders  involved  in 
responding  to  an  emergency  (3)(4)(5).  Adding  to  the  complexity  of  response  operations  is  the  high 
likelihood  that  the  lead-department  will  potentially  change  depending  on  the  specific  nature  of  the 
evolving  safety  or  security  threat.  For  example,  during  the  2005  London  train  bombings,  leadership  was 
one  of  the  primary  issues  reported  after  the  response  had  concluded  (6).  The  uncertainty  surrounding 
leadership  often  results  in  the  blurring  of  existing  boundaries  between  organizations  that  has  long  been 
recognized  to  occur  during  emergency  response  operations  (7),  adding  confusion  to  the  response. 

Many  studies  indicate  the  importance  of  establishing  communication  protocols  and  point  of  contact  lists 
prior  to  an  emergency  (7).  Understanding  these  current  business  practices  is  the  starting  point  for  process 
mapping.  Modelling,  simulation  and  exercises  offer  a  valuable  means  to  explore  procedural,  organization 
and  technology  innovation.  Harbours  provide  an  ideal  setting  for  the  study  of  interdepartmental  safety 
and  security  as  there  is  often  an  intersection  of  land  and  sea,  and  both  public  and  private  domains. 

Process  mapping  was  used  during  the  IHSC  to  document  existing  practices,  provide  insight  and  enhance 
awareness  of  interdepartmental  issues.  The  process  mapping  completed  for  this  project  helped  to 
characterize  challenges  and  suggest  technical  and  procedural  improvements.  To  maximize  the  limited 
time  available  for  data  collection  from  the  participating  agencies,  a  questionnaire  was  developed  and 


distributed  along  with  a  scenario  synopsis.  These  provided  a  base  template  to  collect  the  information 
required  for  the  process  mapping,  such  as: 

•  Roles  and  responsibilities  of  the  departments  and  agencies  involved; 

•  Current  operational  policies  and  operating  procedures  for  the  participating 
departments/ agencies ; 

•  Communication  and  information  sharing  mechanisms,  including  interdepartmental  or 
inter-agency  agreements; 

•  Relevant  information  available  to  share  with  other  departments/agencies,  including 
access  to  databases  or  other  sources  of  information; 

•  Level  and  ease  of  accessibility  of  critical  information; 

•  Information  gaps; 

•  Factors  affecting  timely  access  of  information  and  access  to  critical  resources; 

•  Emergency  management  procedures,  either  as  a  provider  of  critical  information,  or 
responding  agency; 

•  Decision-making  process  for  the  handling  of  the  emergency;  and 

•  Special  communication  procedures  during  the  emergency. 

2.4.  Metric  Development 

Emergency  management  requires  interagency  communication  and  coordination  (7).  The  goal  of  the 
metrics  development  for  the  IHSC  was  to  identify  the  key  performance  indicators  that  describe 
interdepartmental  response,  and  to  benchmark  their  value  through  effective  observation  and  measurement. 
The  evaluation  was  based  on  operator’s  subjective  opinion  due  to  constraints  on  evaluator  interactions 
with  exercise  participants,  and  data  recording  restrictions  imposed  at  certain  operation  centers. 

A  series  of  evaluation  tools  were  developed  to  assess  the  performance  in  key  response  areas.  The 
components  contained  in  each  of  the  evaluation  tools  were  determined  through  review  of  the  guidance 
provided  by  the  HSEEP,  and  discussion  with  subject  matter  experts  with  experience  in  each  respective 
area  of  the  evaluation.  Whenever  possible,  the  focus  was  on  the  development  of  Scenario  Independent 
Metrics  (SIM)  that  could  be  applied  to  a  wide  range  of  future  exercises.  Existing  tools  to  measure 
situational  awareness,  such  as  the  Situational  Awareness  Rating  Technique  (SART)  and  the  Situational 
Awareness  Global  Assessment  Technique  (SAG AT)  were  reviewed  prior  to  developing  the  evaluation.  It 
was  deemed  that  the  SAGAT  assessment  was  not  applicable  because  of  the  requirement  to  stop  exercise 
play  to  administer  the  test  (8).  The  SART  on  the  other  hand  provided  a  good  example  of  an  easy-to-use 
test  that  can  be  administered  in  a  wide  range  of  task  types  (8).  Breton  et  al.  provide  an  excellent  critical 
evaluation  of  the  available  metrics  and  tools  to  measure  individual  and  team  situational  awareness  and 
their  applicability  in  Command  and  Control  (C2)  environments  (9). 

Admittedly,  the  SART  provided  the  inspiration  for  the  look  and  feel  of  the  evaluation  forms  developed 
for  the  IHSC,  though,  it  was  not  directly  applicable.  The  SART  is  operator  oriented  and  the  evaluation 
for  the  live  exercise  was  focused  more  on  evaluating  interdepartmental  processes,  not  operator 
performance.  Therefore,  several  new  evaluation  tools  were  developed  for  evaluating  interdepartmental 
process.  The  overall  evaluation  addressed  the  performance,  effectiveness,  efficiency,  and  complexity  of 
the  multi-agency  emergency  response,  including  the  interoperability  between  agencies  in  the  following 
areas: 


•  Command  and  Control  (priorities,  command  structure,  reporting,  documentation,  roles 
and  responsibilities) 

•  Communications  (internal  /  external  and  formal  /  informal) 


•  Interdepartmental  Information  Sharing  (unclassified) 

Evaluation  forms  were  chosen  as  the  primary  method  of  data  collection  during  the  live  exercise. 
Evaluation  forms  were  selected  because  they  provide  structure  to  data  collection  and  observation  at  each 
of  the  exercise  evaluation  locations,  and  are  easy  to  administer  with  little  obtrusiveness.  Forms  are  also 
easy  to  circulate  prior  to  an  exercise  for  approval  by  the  participating  agencies  and  departments.  Four 
evaluation  tools  were  developed,  namely:  Operations  Synchronization  Assessment  Tool  (OS AT), 
Communications  Assessment  Tool  (CAT),  Post  Exercise  Player  Assessment  (PEP A),  and  the 
Coordination  Group  Assessment  (CGA).  A  5-point  scale  numeric  scale  and  comment  boxes  were  used  in 
all  assessments  to  capture  player  and  subject  matter  expert  opinion. 

2.4.1.  Data  Collection 

Eleven  agencies  from  the  municipal,  provincial,  and  federal  levels  agreed  to  participate  in  the  live  excise 
evaluation  component  of  the  IHSC.  Exercise  evaluators  were  given  permission  to  administer  the 
assessment  forms  to  the  players  and  controllers  participating  in  the  exercise.  Effort  was  made  to  examine 
the  MSEL  to  ensure  the  optimum  placement  of  evaluators  to  maximize  the  amount  of  collected  data.  The 
primary  consideration  was  how  involved  the  agency  was  in  the  scenario  and  whether  or  not  they  were  a 
key  operational  node. 

Several  constraints  were  observed  during  the  data  collection  phase  of  exercise.  Some  agencies  chose  to 
limit  the  number  of  interactions  that  their  players  and  controllers  could  have  with  the  evaluators  during 
the  exercise.  This  affected  the  distribution  of  responses  received  from  each  of  the  individual  agencies. 
The  results  presented  in  this  paper  represent  the  aggregate  of  all  received  responses.  No  effort  has  been 
made  to  normalize  the  number  of  responses  used  from  each  agency  to  ensure  the  results  of  an  individual 
agency  are  not  over-represented.  For  each  assessment,  a  dashboard  overview  of  the  results  will  be  given 
containing  results  rounded  to  the  nearest  quarter.  Table  1  contains  a  summary  of  the  survey  responses. 

Table  1:  Summary  of  Evaluation  Responses 


Category 

Total 

Number  of  Locations 

11 

Operations  Synchronization  Assessment  Test  (OSAT) 

38  responses 

Communications  Assessment  Test  (CAT) 

26  responses 

Coordination  Group  Assessment  (CGA) 

10  responses 

Post-Exercise  Player  Assessment  (PEPA) 

32  responses 

2.4.2.  Operations  Synchronization  Assessment  Tool  (OSAT) 

The  OSAT  formed  the  primary  evaluation  of  command  and  control  and  information  sharing.  To  act 
effectively  in  an  emergency  requires  sharing  and  using  information  effectively:  collecting,  collating, 
analyzing,  and  then  deploying  it  promptly  in  a  useful  form.  The  focus  of  the  OSAT  was  on  the  operational 
synchronization  of  the  eleven  agencies  participating  in  the  exercise  evaluation. 

The  purpose  of  establishing  some  form  of  operational  synchronization  is  the  maintenance  of  synchronized 
activity  and  process  among  distributed  parties.  It  is  most  critical  in  rapidly  evolving  situations  or  in 
highly  distributed  operations  (10).  The  emergency  response  associated  with  the  live  exercise  in  Halifax 
Harbour  was  both  highly  distributed  and  rapidly  evolving.  For  this  reason  the  OSAT  was  seen  as  a  key 
assessment  tool  that  needed  to  be  developed  for  the  exercise.  Successful  battle  rhythm  implies  the 
synergism  of  procedures,  processes,  technologies,  individual  activities  and  collective  action  at  all  levels  in 
order  to  facilitate  the  emergency  response  (10).  The  OSAT  was  designed  to  explore  the  level  of 


synchronization  of  planning,  monitoring,  and  reporting  between  the  agencies  and  departments 
participating  in  the  assessment. 

Valid  and  timely  information  sharing  is  critical  during  emergency  response  operations  (7).  It  also  directly 
impacts  the  level  of  synchronization  achieved  between  responding  agencies.  A  significant  source  of 
confusion  during  rapidly  evolving  emergency  response  operations  is  information  ambiguity  or  conflicts. 
This  was  directly  observed  during  the  live  exercise.  Conflicting  event  reports  being  shared  between  the 
participating  agencies  during  the  exercise  led  to  a  significant  divergence  in  planning  synchronization. 
Table  2  provides  an  overview  of  the  OSAT  assessment  tool. 

Table  2:  Overview  of  the  components  and  criteria  used  to  assess  Operations  Synchronization 


Concept 

Component 

Metric 

Description 

Involvement  in  Decision 
Making 

level  of  involvement  of  your  agency  in 
operational  level  decision  making 

Planning 

Synchronization 

Knowledge  of  Operational 
Priorities 

Degree  to  which  the  current  operational 
priorities  are  known  to  your  agency 

Clarity  of  Responsibilities 

Degree  to  which  the  responsibilities  of  your 
agency  are  known  and  understood 

Understanding  of 
situation 

Level  of  understanding  of  the  situation 

Operations 

Synchronization 

Monitoring 

Synchronization 

Stability  of  situation 

Situations  degree  of  stability  with  respect  to  the 
overall  emergency  response 

Evolution  of  situation 

Ability  to  anticipate  future  required  actions  to 
control  the  situation 

Clarity  of  Lead  Agency 
Reporting  Expectations 

Clarity  of  the  reporting  expectations  of  the  lead 
department.  Requirements  include:  Type  and 
format  of  information,  and  reporting  schedule 

Reporting 

Synchronization 

Reporting  Cycle  Demand 

Ability  to  meet  reporting  requirements  of  the 
lead  agency 

Appropriateness  of  Report 
Demands 

Degree  to  which  lead  agency's  reporting 
expectations  utilize  your  agency 

The  OSAT  assessment  was  dependent  upon  the  existence  of  a  lead  agency  utilizing  the  capabilities  of 
multiple  supporting  agencies.  The  low  numbers  in  the  reporting  category  are  to  be  expected  from  an 
exercise  of  only  10  hours  duration  (see  Figure  2).  Even  if  there  are  clear  reporting  requirements  and 
processes  in  place  to  support  operations  synchronization,  they  are  by  definition  time-based  cyclical 
requirements  that  take  some  time  to  get  established  and  become  routine.  The  average  response  value  for 
knowledge  of  operational  priorities  and  understanding  of  the  situation  during  the  exercise  were  fairly 
high.  During  a  post  exercise  hot  wash,  this  was  discussed  and  attributed  to  the  fact  that  most 
organizations  were  simply  executing  their  normal  emergency  mandates,  requiring  little  interpretation  of 
the  situation. 


Figure  2:  Operations  Synchronization  Assessment  Test  (OSAT)  Dashboard  (N  =  36) 


2.4.3.  Communications  Assessment  Tool  (CAT) 

The  communications  assessment  component  of  the  evaluation  looked  at  the  interoperability,  performance 
and  effectiveness  of  the  communications  systems  in  use  during  the  exercise,  and  the  communications 
environment.  According  to  the  U.S.  Department  of  Homeland  Security,  $2.15  billion  in  grant  funding 
was  awarded  to  states  and  localities  from  2003  to  2005  for  communications  interoperability 
enhancements  (5). 

In  Canada,  several  initiatives  exist  to  enhance  interoperability  of  emergency  responders.  Current 
initiatives  include  the  development  of  Canadian  National  Information  Exchange  Model-Based 
Architectures  (C-NIEM)3,  Canadian  Profile  of  the  National  Common  Alerting  and  Notification  Protocol 
(CAP-CP)4,  and  the  establishment  of  the  Canadian  Interoperability  Technology  Interest  Group  (CITIG)5. 
Much  work  has  also  been  done  to  improve  exercise  evaluation  and  improvement  planning.  HSEEP 
provided  excellent  guidance  for  the  development  of  metrics  for  exercise  evaluation,  including  the 
assessment  of  communications  (2).  Table  3  contains  an  overview  of  the  communications  assessment  tool. 


3  The  U.S.  National  Information  Exchange  Model  (NIEM)  is  an  XML-based  data  exchange  standard  as  well  as  a 
common  lexicon  that  was  developed  by  a  partnership  between  the  U.S.  Department  of  Justice  (DOJ)  and  the  U.S. 
DHS.  C-NIEM  is  a  Canadian  implementation  of  NIEM. 

4  The  Common  Alerting  Protocol  (CAP)  is  an  XML-based  data  format  that  was  developed  in  the  United  States  (US) 
for  the  exchange  of  public  warnings  and  emergency  alerts  between  alerting  technologies. 

5  The  Canadian  Interoperability  Technology  Interest  Group  (CITIG)  is  a  partnership  between  Canadian  Police 
Research  Center  and  other  key  first  responder  associations  promoting  interoperability  by  bringing  together  the 
collective  wisdom  of  first  responder  and  communication  leaders  and  experts. 


Table  3:  Overview  of  the  components  and  criteria  used  to  assess  communications 


Concept 

Component 

Metric 

Description 

Support  of 

Information 

Requirements 

The  communication  systems  in  use  during  the  event 
adequately  supported  your  critical  information  needs 

Communications 

Performance 

Support  of 

Situational 

Awareness 

The  communication  systems  in  use  during  the  event 
adequately  supported  your  situational  awareness 

Support  of 

Operations 

Synchronization 

The  communication  systems  in  use  during  the  event 
adequately  supported  operations  synchronization 

Effective 

Communications 

Level  of  Clutter 

Degree  of  redundant,  irrelevant,  or  inappropriate 
communications 

Communications 

Effectiveness 

Level  of  Clarity 

Were  received  communication  transmissions  easily 
interpreted  and  understood? 

Level  of  Timeliness 

Degree  of  latency  in  the  communication  system  for  a 
response  to  a  request 

Communications 

Environment 

Communications 
System  Availability 

Degree  of  system  availability  to  the  operator.  This  could 
include:  number  handsets  available  to  the  operator,  or 
channel  availability 

Communications 

Environment 

Degree  that  the  communications  environment  supports 
communications  activity.  This  includes:  Ambient  noise, 
head-set  availability,  loud-speaker  availability,  room 
interference,  communications  system  location  relative 
to  work  station,  etc. 

The  results  of  communications  assessment  relate  to  the  effectiveness  and  performance  of  the 
communications  systems  in  use  during  the  live  exercise.  The  overall  results  of  this  assessment  show  that 
most  responders  were  satisfied  with  the  communications  during  the  exercise  (see  Figure  3).  The  lowest 
scores  were  in  the  communications  effectiveness  category.  The  challenge  with  this  category  is  that  it  is 
heavily  impacted  by  exercise  artificiality.  The  use  of  alternative  frequencies  and  non-standard 
communications  equipment  may  have  contributed  to  the  low  scores  in  this  category.  It  would  have  been 
useful  to  ask  exercise  participants  to  comment  on  the  impact  alternative  frequencies  and  non-standard 
equipment  had  on  their  ability  to  effectively  communicate  in  order  to  better  interpret  these  results. 


COMAS 

Category 

Question 

Average  Response  Value 

Comms 

Performance 

Support  of 

Information 

Requirements 

3.3  (+-  1.1) 

Support  of  Situational 

Awareness 

3.1  (+-  1.1 

Support  of 

Operations 

Synchronization 

3.0  (+- 1.2) 

Comms 

Effectiveness 

Note:  Lower 
score  is  worse 
in  this  category 

Level  of  Clutter 

2.7  (+-  0.9) 

Level  of  Clarity 

3.2  (+- 1.0) 

Level  of  Timeliness 

2.9  (+- 1.0) 

Comms 

Environment 

Communications 

System  Availability 

3.7  (+- 1.0) 

Communications 

Environment 

3.3  (+-  0.9) 

Numeric  Value  1  2 

Response  Very  Low  Low 


4  5 

High  Very  High 


Figure  3:  Dashboard  overview  of  CAT  results  (N  =  26) 
2.4.4.  Coordination  Group  Assessment  (CGA) 


The  focus  of  the  Coordination  Group  Assessment  (CGA)  was  on  the  form  and  function  of  the  Federal 
Coordination  Group6  (FCG)  and  the  Federal  Coordination  Steering  Committee7  (FCSC)  (1 1).  FCSC 
membership  reflects  the  emergency  support  functions  required  to  respond  to  a  given  emergency. 
However,  the  composition  of  the  FCSC  membership  is  flexible  in  order  to  accommodate  specific  threats 
and  hazards  as  they  arise.  For  instance,  the  membership  of  the  FCSC  may  be  limited  to  security  and 
intelligence  stakeholders  if  a  national  security  threat  is  perceived  or  determined  to  exist  to  ensure 
Operational  Security  (OPSEC).  Other  members  would  then  be  engaged  as  appropriate  to  provide 
assistance  and  advice  to  deal  with  resulting  contingencies  from  the  initial  threat.  The  FCSC  will  make  a 
collective  risk-management  recommendation  to  engage  a  broader  community,  including  provincial  and 
territorial  partners,  when  consequence  management  arrangements  are  required  to  be  engaged.  This 
decision  will  serve  as  the  initial  activation  of  the  National  Emergency  Response  System  (NERS).  Figure 
4  provides  an  overview  of  the  federal  regional  governance  within  the  NERS  construct. 


6  The  Federal  Coordination  Group  (FCG)  is  a  standing  committee  composed  of  emergency  management  managers 
from  federal  departments  in  the  region.  The  group  reports  to  the  Federal  Coordination  Steering  Committee.  The 
Public  Safety  Canada  Regional  Director  co-chairs  this  group.  During  an  emergency,  the  Federal  Coordination  Group 
provides  emergency  management  planning  and  advice.  It  also  provides  and/or  manages  the  flow  of  information  and 
requests  for  federal  assistance  within  the  region. 

7  The  Federal  Coordination  Steering  Committee  (FCSC)  is  a  steering  committee  composed  of  senior  regional 
departmental  representatives.  The  Public  Safety  Canada  Regional  Director  co-chairs  this  committee.  The  Committee 
provides  direction  on  emergency  management  planning  and  preparedness  activities.  It  also  oversees  the  coordination 
of  the  federal  regional  response. 


The  FCG,  as  per  its  mandate  outlined  in  the  FERP,  will  conduct  the  planning  and  implementation  of  the 
federal  regional  response  to  an  event  in  accordance  with  direction  provided  by  the  FCSC.  PS  acts  as  co¬ 
chair  on  both  of  these  committees  to  support  the  lead  agency  and  to  enable  effective  information  sharing 
and  situational  awareness  with  regards  to  the  emergency  response.  Departments  may  receive  information 
as  a  result  of  FCSC/FCG  meetings  that  could  trigger  the  activation  of  their  Emergency  Operations 
Centers  (EOC),  but  this  remains  an  internal  departmental  decision.  A  recommendation  may  be  made  by 
the  FCSC/FCG  to  activate  an  EOC,  but  formal  direction  to  activate  EOCs  does  not  come  from  the 
FCSC/FCG.  This  recommendation  is  based,  in  part,  on  considerations  for  OPSEC  and  command  and 
control.  Tactical  operations  continue  to  be  conducted  at  the  direction/discretion  of  the  individual 
departments.  The  plan  created  by  the  FCG  is  meant  to  guide  tactical  activity  in  such  a  manner  as  to 
ensure  that  operational  and  strategic  goals  of  the  overall  response  are  met. 
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Federal  Government  Provincial  Government 


1  Prime  Minister 

- . - -► 
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Figure  4:  Overview  of  federal  regional  governance  within  the  NERS8 


The  questions  posed  in  the  CGA  examined  the  structure  of  the  FCG  and  FCSC  and  how  they  handle  the 
coordination  of  regional  federal  actions.  Following  each  convening  of  the  FCG,  or  the  FCSC,  the  CGA 
was  administered  to  each  participant  in  the  meeting.  The  evaluation  was  based  on  participant’s  subjective 
opinion  on  the  structure  of  the  meetings  and  their  ability  to  effectively  make  decisions.  As  described 
earlier,  a  five  point  scale  was  used  to  measure  their  response.  Table  4  contains  an  overview  of  the  CGA 
assessment  tool. 


Federal  regional  governance  flowchart  provided  in  part  by  the  Nova  Scotia  regional  office  of  Public  Safety 


Canada. 


Table  4:  Overview  of  the  components  and  criteria  forming  the  Coordination  Group  Assessment 


Topic 

Component 

Criteria 

Structure 

The  optimum  group  of  agency  representatives  were  assembled  to  coordinate  the  overall 
response 

The  format  of  the  meeting  (participants,  information  sources,  and  required  actions)  was  well 
defined  prior  to  the  event 

Coordination 

Technical  and/or  science-based  advice  was  available  as  required? 

Decision 

Additional  community  members  were  engaged  as  appropriate  to  provide  assistance  and  advice 
on  the  situation? 

Making 

Overall,  the  critical  information  required  to  coordinate  the  regional  federal  resources  and 
overall  response  activities  was  available? 

Overall,  the  coordination  group  was  able  to  carry  out  their  primary  functional  responsibilities? 

The  results  of  the  coordination  group  assessment  indicate  that  there  was  good  support  for  the  current 
structure  and  process,  but  that  the  group’s  situational  awareness  and  access  to  the  right  external  expertise 
during  the  coordination  group  meetings  was  low  (see  Figure  5).  Improving  situational  awareness  and 
access  to  external  expertise  are  ideal  candidates  for  technology  solutions.  Engaging  additional 
community  members  for  advice  during  the  coordination  group  meetings  received  the  lowest  score  in  the 
assessment.  This  would  indicate  that  future  efforts  should  focus  on  technology  solutions  that  would 
increase  access  to  relevant  community  members  as  required. 


CGA 


Description 


Overall  the  critical  information 
required  to  coordinate  the  regional 
federal  resources  and  emergency 
response  activities  was  available? 


T e  chnic  al  and  or  scienc  e-b  a  se  d 
advice  was  available  as  required? 


Additional  community  members 
were  engaged  as  appropriate  to 
provide  assistance  and  advice  on 
the  situation? 


The  optimum  group  of  agency 
representatives  was  assembledto 
coordinate  the  emergency 
response? 


The  format  o  f  the  meeting 
(participants,  information  sources, 
and  required  actions)  was  well 
definedprior  to  the  event? 


Overall,  the  coordination  group 
was  able  to  cany  out  their" primary 
functional  re  sp  onsibilitie  s 


Average  Response  Value 


3.1  (+-  0.7) 


3.0  (+-  0.8) 


2.4  (+-  1.4) 


4.0  (+-  0.9) 


4.0  (+-  0.9) 


4.0  (+-  0.7) 


Numeric  Value  1  2 

Response  Very  Low  Low 


4  5 

High  Very  High 


Figure  5:  Dashboard  results  for  the  CGA  (N  =  10) 


2.4.5.  Post  Exercise  Player  Assessment  (PEP A) 


The  Post  Exercise  Player  Assessment  (PEP A)  was  used  to  assess  player  satisfaction  in  a  number  of  key 
areas.  The  questions  asked  in  the  PEPA  can  be  loosely  grouped  into  three  categories  (See  Table  5).  The 
first  category  relates  to  the  communications  tools,  workstation  displays,  and  Points  of  Contact  (POC)  list 
used  during  the  exercise.  The  second  category  relates  to  the  time  required  to  send  and  receive  critical 
information  and  the  ability  to  communicate  with  other  command  centres.  The  last  category  related  to  the 
reliance  of  an  agency  on  informal  communications  to  perform  its  mandate. 


Table  5  :  Overview  of  the  components  and  criteria  forming  the  Post  Exercise  Player  Assessment 


Topic 

Component 

Criteria 

Tools 

Overall,  the  communication  tools  (e.g.  software,  phones,  VTC,  secure  /  unsecure  networks) 
that  I  used  were  adequate  to  support  my  requirements 

My  workstation  displays  provided  adequate  support  to  my  information  requirements 

Overall,  I  had  the  Points  of  Contact  (POC)  I  needed  to  send  information  appropriately 

Communications 

Communication 

Overall,  the  amount  of  time  required  to  transmit  critical  information  was  acceptable 

Connectivity 

Overall,  the  amount  of  time  required  to  receive  critical  information  was  appropriate 

Overall,  communication  with  other  command  centres  and  agencies  was  adequate 

Informal 

Communications 

Overall,  informal  communications  methods  were  important  in  fulfilling  your  duties  and  for 
passing  information 

The  results  of  the  post  exercise  assessment  present  a  relatively  positive  opinion  of  the  tools  the  players 
used  to  perform  their  job  and  their  ability  to  share  information  with  other  agencies  (see  Figure  6). 
Communications  with  other  command  centres  received  the  lowest  score  during  the  post  exercise  player 
assessment.  The  relatively  positive  response  to  the  question  of  having  the  correct  points  of  contact  is 
offset  by  a  commensurate  level  of  use  of  those  contacts  to  support  a  reliance  on  informal  communications 
methods  to  fulfil  information  needs.  This  is  reflected  in  feedback  that  indicated  a  number  of 
organizations  simply  used  informal  methods  to  try  to  build  situational  awareness  in  the  absence  of  a 
common  emergency  management  system.  This  latter  observation  is  puzzling  in  the  context  of  the  equally 
high  response  to  the  second  question,  which  would  indicate  their  current  tool  supported  their  information 
requirements  to  a  relatively  high  degree.  The  response  to  the  second  question  does  not  seem  to  have  been 
supported  by  anecdotal  feedback. 


Description 


Overall  the  communications  tools 
(software.  phones;  VTC.  etc.)  that 
I  used  were  adequate  to  support 
my  requirements'1 


My  work  station  displays  provided 
adequate  support  to  mv 
infonnationrequirements1 


Overall,  I  had  the  Points  of 
Contacts  (POC)  I  needed  to  send 
information  appropriately? 


Overall,  the  amount  oftime 
required  to  transmit  critical 
information  was  acceptable 


Overall,  the  amount  oftime 
require  to  receive  critical 
information  was  acceptable? 


Overall  communications  with 
other  command  centers  and 
agencies  was  adequate1 


Overall,  informal  communication 
methods  were  important  in 
fulfilling  your  duties  and  for 
p  a  ssing  information 


Average  Response  Value 


3.5  (+-  1.0) 


3.8  (+-  0.9) 


3.9  (+-  0.8) 


3.6  (+-  1.1) 


3.3  (+- 1.1) 


3.0  (+-  1.0) 


3.6  (+- 1.2) 


Numeric  Value  1  2 

Response  Very  Low  Low 


4  5 

High  Very  High 


Figure  6:  Dashboard  results  for  the  PEPA  (N  =  32) 


3.  Challenges 
3.1.  General 

The  assessment  of  stakeholder  interoperability  was  identified  as  a  primary  objective  of  the  metrics  and 
evaluation  developed  for  the  IHSC.  Live  exercises  provide  an  excellent  opportunity  to  examine 
command  and  control  procedures,  as  well  as  stakeholder  roles  and  responsibilities.  Direct  observation 
provides  the  basis  for  evidence-based  recommendations,  and  aids  in  the  development  of  lessons  learned. 
Not  all  agencies  and  departments  have  the  same  views  towards  observation-based  assessments. 
Observations  made  during  exercises  could  be  viewed  by  some  as  an  auditing  activity,  which  may  have  a 
negative  connotation.  It  is  important  during  the  early  exercise  planning  stages  to  clearly  articulate  the 
measurement  and  assessment  objectives,  and  ensure  that  they  are  understood  and  agreed  on  by  all 
stakeholders.  Allowing  an  external  observer/evaluator  into  a  normally  closed  operations  center  is  not  a 
trivial  decision,  and  approval  for  such  activity  should  not  be  taken  for  granted. 

The  exercise  brought  together  a  very  diverse  group  of  agencies  and  departments.  Some  agencies  involved 
in  the  exercise  have  never  participated  in  an  interdepartmental  training  exercise,  and  some  had  never  been 
exposed  to  observation  by  an  external  evaluator.  In  this  sense,  the  exercise  provided  many  firsts  for  some 
of  the  participating  agencies.  Prior  to  deploying  evaluators  during  the  exercise  a  significant  effort  was 
required  to  develop  a  relationship  with  the  participating  agencies  and  departments  to  ensure  they 
understood  the  purpose  of  having  an  evaluator  at  their  location.  These  relationships  form  the  backbone  of 
an  effective  evaluation  plan  and  should  be  fostered  as  early  as  possible  in  the  exercise  planning.  Access 
to  the  right  place  at  the  right  time  is  crucial  in  order  to  conduct  an  effective  evaluation  during  a  live 
exercise,  and  is  not  possible  without  the  explicit  support  of  the  participating  agencies. 


3.2.  Process  Mapping 


Part  of  the  IHSC  focused  on  the  collection  of  operating  procedures  and  practices  for  the  harbour  of 
Halifax  to  produce  a  harbour  security  process  map.  Process  mapping  has  been  successfully  used  to 
examine  existing  interdepartmental  processes  and  to  provide  a  starting  point  for  process  improvements. 
Halifax  Harbour  provided  a  very  challenging  scenario  for  harbour  security  process  mapping.  Challenges 
encountered  were  the  multitude  of  stakeholders,  the  potential  for  overlapping  jurisdictions,  and  the 
existence  of  draft  interdepartmental  emergency  standard  operating  procedures. 

Process  mapping  is  in  some  cases  too  rigid  to  handle  the  rapidly  evolving  non-linear  world  of  emergency 
management.  Routine  work  is  characterized  by  a  linear  and  sequential  process  of  conversion  of  inputs 
into  outputs.  Process  mapping  excels  in  describing  these  situations  where  the  relationship  between  inputs 
and  outputs  are  both  well  known  and  understood.  However,  multi-agency  emergency  response  represents 
non-routine  work,  which  is  characterized  by  multiple,  concurrent  and  non-linear  processes,  where  issues 
cannot  be  resolved  simply  by  being  plugged  into  an  elegant  model  or  established  procedures  (12). 

Standard  operating  procedures  for  emergency  management  represent  fixed  procedure,  but  are  difficult  to 
define  for  all  possible  ranges  of  events.  Management  actions  are  scenario  dependant  and  can  be 
characterized  by  several  parameters  with  potential  non-linear  interactions  such  as,  the  nature  of  the 
emergency,  geographic  location,  and  available  resources.  For  a  fixed  scenario  covered  by  well  defined 
standard  operating  procedures,  a  process  map  can  be  produced.  If  the  scenario  changes  and  falls  outside 
the  scope  of  the  standard  operating  procedure,  the  process  map  will  not  necessarily  remain  valid. 

Producing  a  process  map  for  a  live  exercise  is  greatly  aided  by  the  development  of  a  detailed  MSEL.  A 
MSEL  provides  a  timeline  of  events  for  the  live  exercise  and  often  contains  event  triggers  and  expected 
actions.  Expected  actions  are  actions  assigned  to  a  specific  agency,  based  on  Standard  Operating 
Procedures  (SOP),  Service  Level  Agreements  (SLA),  and  Memorandum  of  Understanding  (MOU),  in 
response  to  an  event  contained  in  the  MSEL.  The  expected  actions  in  a  MSEL  are  often  based  on  expert 
judgment  during  the  planning  sessions  when  the  MSEL  is  created. 

The  MSEL  produced  by  PS  was  instrumental  in  the  construction  of  a  process  map  of  the  live  exercise. 
Unfortunately,  as  the  MSEL  is  periodically  updated  to  reflect  changes  to  the  exercise,  the  process  map 
must  also  be  updated  to  remain  valid.  The  MSEL  provided  information  on  roles  and  responsibilities, 
existing  operating  procedures,  communication  and  information  sharing  mechanisms,  and  emergency 
management  and  decision  making  function  during  the  exercise.  A  good  MSEL  provides  an  “As  Is” 
process  map  as  understood  by  the  SME  representing  the  participating  agencies  and  departments. 

3.3.  Metrics 

As  stated  earlier,  it  is  of  the  utmost  importance  to  clearly  articulate  the  measurement  and  assessment 
objectives  during  the  early  exercise  planning  stages,  and  to  ensure  that  they  are  understood  and  agreed  on 
by  all  stakeholders.  Often  data  collection  and  observation  becomes  a  low  priority  in  a  hectic  operations 
center  during  a  live  exercise.  This  was  indeed  the  case  at  several  locations  where  observers  were 
deployed  during  the  live  exercise.  The  impact  of  this  was  minimized  by  limiting  interactions  with 
exercise  players  and  instead  focusing  on  SME  who  were  not  directly  involved  in  the  live  exercise.  The 
SME  were  knowledgeable  of  the  roles  and  responsibilities  of  the  agency/department,  and  were  well 
versed  in  the  standard  operating  procedures  being  used  by  the  exercise  participant.  The  SME  were  also 
typically  the  internal  evaluator  of  the  given  agency  and  therefore  were  fully  aware  of  the  exercise  events 
and  relevant  internal  agency  processes  to  be  exercised.  The  pairing  of  an  evaluator  with  a  SME  is  ideal  in 
an  operations  center  where  direct  access  to  an  exercise  player  will  be  limited  or  discouraged. 


The  development  of  evaluation  tools  that  are  designed  to  assess  multi-agency  interoperability  comes  with 
its  own  unique  set  of  challenges.  Evaluation  tools  must  not  be  too  specific,  as  many  agencies  are 
involved  and  do  not  come  from  the  same  backgrounds  (i.e.  civil  or  military).  Concepts  and  wording 
contained  in  the  evaluation  tools  must  not  be  domain  specific  to  ensure  that  the  results  can  be  compared 
across  stakeholders. 

Producing  a  holistic  assessment  posed  a  significant  challenge  for  the  IHSC  given  the  diversity  of 
participating  agencies  and  departments.  This  is  a  fact  of  life  in  a  busy  harbour  setting  where  many  public 
and  private  interests  intersect  and  jurisdictions  may  overlap.  The  evaluation  tools  deployed  during  the 
IHSC  attempted  to  focus  on  general  concepts  that  were  common  to  the  community  of  stakeholders. 

Effort  was  made  to  identify  concepts,  components,  and  metrics  that  would  be  understood  by  all 
stakeholders  participating  in  the  live  exercise. 

A  one  day  exercise  that  must  finish  on  schedule  does  not  allow  for  rigorous  time -based  assessments  of 
interoperability.  The  time  from  when  an  event  is  triggered  to  when  the  expected  response  action  must 
occur  is  pre-determined.  The  live  exercise  had  a  significant  amount  of  free  play,  but  it  still  had  to  follow 
a  schedule  wherein  exercise  controllers  were  responsible  to  force  expected  actions  in  accordance  with  the 
MSEL  timeline.  This  artificiality  makes  the  interpretation  of  rigorous  time -based  observations  difficult 
to  interpret  and  of  limited  use.  Artificiality  also  affected  the  assessment  of  interdepartmental  decision 
making  and  information  sharing  since  the  output  of  these  activities  had  to  progress  in  accordance  with  the 
MSEL  timeline. 

Information  sharing  is  greatly  influenced  by  existing  relationships  when  time  is  a  factor,  often 
contradicting  expected  information  flow.  Early  during  the  development  of  metrics  for  the  IHSC, 
information  tracking  formed  a  signification  portion  of  the  assessment.  Emphasis  was  later  shifted  away 
from  tracking  information  because  of  the  impact  it  would  have  on  the  exercise  participant  and  the 
demands  it  would  place  on  the  observer.  Since  a  decision  was  made  during  the  IHSC  to  restrict  the 
recording  of  video,  voice,  or  digital  information  during  the  live  exercise  it  would  be  a  significant 
challenge  for  them  to  track  information  exchanges  in  a  busy  operations  center.  In  an  exercise  where 
recording  of  video,  voice,  and  digital  information  are  permitted,  exercise  reconstruction  and  play  back 
could  enable  information  tracking  post  exercise. 

3.4.  Future  Work 

In  the  future,  more  effort  will  be  made  to  establish  common  performance  measurement  goals  among 
stakeholders  in  Harbour  security  prior  to  future  exercises.  The  assessment  completed  as  part  of  the  IHSC 
did  not  focus  on  establishing  common  measurement  goals,  but  may  have  benefited  greatly  if  it  had.  As  an 
example,  if  a  common  measurement  goal  had  been  to  evaluate  the  effectiveness  of  existing  SLA,  then  the 
evaluation  would  focus  on  identifying  SLA  between  agencies  and  evaluating  their  ability  to  support 
existing  requirements.  This  would  enable  a  conventional  approach  to  developing  metrics  that  derives 
from  a  problem  statement  and  subsequent  identification  of  determinant  factors  and  key  performance 
parameters. 

Another  potential  area  for  study  is  the  development  of  self-reporting  standards  for  harbour  security  live 
exercises.  Establishing  self-reporting  standards  would  promote  more  effective  data  management  and 
allow  agencies  to  better  track  and  report  their  own  progress,  eliminating  the  requirement  for  external 
evaluators  to  access  sensitive  locations  during  an  exercise,  as  discussed  in  3.1.  This  would  also  result  in 
a  reduction  in  the  number  of  individuals  required  to  be  present  at  a  given  exercise  location  which,  based 
on  experience,  would  be  viewed  as  significant  benefit. 


4.  Conclusions 


Most  emergencies  in  Canada  are  local  in  nature  and  are  managed  at  the  municipal  or  provincial/territorial 
level.  However,  certain  risk  factors  increase  the  potential  for  catastrophes  to  transcend  geographical 
boundaries  and  to  challenge  the  capacity  of  federal  and  provincial/territorial  governments  to  manage 
emergencies(l  1).  Harbour  security  challenges  existing  capacity  because  of  the  multiplicity  of  threats  - 
air,  land  and  water,  and  because  of  the  potential  for  overlapping  jurisdictions  within  the  harbour 
environment.  The  IHSC  provided  the  first  assessment  of  interdepartmental  process  of  this  type  in  Halifax 
Harbour.  The  live  exercise  provided  a  useful  forum  for  stakeholders  at  all  levels  to  exercise  their  capacity 
to  respond  effectively  to  marine -based  safety  and  security  incidents. 

Given  the  large  number  of  participants  involved  in  marine  safety  and  security  in  Halifax  Harbour,  several 
key  challenges  associated  with  developing  a  comprehensive  evaluation  where  encountered,  as  described 
in  section  3.  The  process  mapping  activity  associated  with  the  IHSC  yielded  several  interesting 
observations  during  its  completion.  The  first  is  that  the  initial  planning  conference  (IPC),  mid  planning 
conference  (MPC),  and  the  final  planning  conference  (FPC),  where  the  exercise  design  team  and 
stakeholders  gather,  provides  an  excellent  venue  to  answer  questions  and  resolve  ambiguities  relating  to 
information  gaps  associated  with  process  mapping. 

Planning  conferences  are  made  up  of  SME  from  each  respective  stakeholder  that  are  often  very  adept  at 
answering  questions  relating  to  interdepartmental  processes.  These  same  individuals  are  also  ideal  to  be 
paired  with  exercise  observers  during  a  live  exercise  to  help  limit  the  number  of  interactions  with  busy 
exercise  participants.  They  often  provide  context  during  observation  and  have  a  better  understanding  of 
the  bigger  picture  as  it  relates  to  their  organization.  Second,  the  MSEL  provides  an  excellent  starting 
point  for  process  modeling  associated  with  a  live  exercise.  One  note  of  caution  is  that  this  activity  should 
only  be  initiated  once  the  MSEL  is  sufficiently  mature.  An  exercise  MSEL  may  be  developed  through 
several  iterations,  forcing  changes  to  existing  process  models  to  reflect  the  changes. 

Similar  to  the  observations  made  during  the  process  modeling  completed  for  the  IHSC,  the  development 
of  metrics  also  yielded  several  interesting  observations.  The  first  is  that  if  metrics  and  the  subsequent 
evaluation  are  to  form  an  integral  part  of  a  live  exercise  they  must  be  presented  as  early  as  possible  in  the 
planning  process.  Experience  gained  during  the  IHSC  highlight  the  importance  of  fostering  relationships 
with  all  stakeholders  to  ensure  that  data  collection  and  observation  does  not  become  a  low  priority  on  the 
day  of  the  exercise.  Also,  effort  should  be  made  to  try  and  produce  SIM  for  use  during  a  live  exercise  to 
promote  reusability  and  potentially  increase  continuity  between  exercises.  Due  to  limited  training 
resources  the  same  emergency  scenario  is  almost  never  used  twice.  The  lack  of  commonality  between 
exercises  makes  the  comparison  of  results  difficult.  SIM  offer  one  opportunity  to  increase  the  continuity 
between  emergency  management  exercises. 

The  IHSC  and  the  live  exercise  designed  and  led  by  PS  represent  a  part  of  the  investment  by  the 
Government  of  Canada  to  improve  marine  security  in  Canada’s  territorial  waters  and  shore  facilities.  The 
results  of  these  two  investments  will  be  used  by  individual  exercise  stakeholders  to  improve  their  internal 
processes  and  by  the  broader  S&T  community  to  help  enhance  harbour  security  in  Canada. 
Interoperability  is  not  something  you  want  to  create  during  emergencies;  it  must  become  part  of  steady 
state  normal  operations.  The  relationships  formed  during  the  IHSC  and  participation  in  the  live  exercise 
will  serve  to  improve  interoperability  and  emergency  management  in  Halifax  Harbour. 
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Overview 


•  Marine  security,  specifically  port  and  harbour  security, 
has  been  identified  as  one  of  the  Government  of 
Canada’s  priorities 

•  Harbour  security  is  a  complex  problem  because  of  the 
multiplicity  of  threats  -  air,  land  and  water,  and  because 
of  the  potential  for  overlapping  jurisdictions  within  the 
harbour  environment 


In  response,  Defence  Research  and  Development  Canada 
-  Atlantic  examined  existing  inter-agency  processes  in  a 
major  harbour  and  developed  metrics  for  use  during  a 
live  multi-agency  harbour  safety  and  security  exercise 
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Common  Marine  Security/Safety  Examples 


Human  Trafficking 


Waterside  Attacks 
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Metrics  Development 


•  The  goal  of  the  metrics  development  for  the  IHSC 
was  to  identify  the  key  performance  indicators  that 
provide  insight  into  interdepartmental  emergency 
response,  and  not  simply  numbers 

•  The  evaluation  was  based  on  operator’s  subjective 
opinion  due  to  constraints  on  evaluator  interactions 
with  exercise  participants,  and  data  recording 
restrictions  imposed  at  certain  operation  centers. 

•  The  focus  was  on  the  development  of  Scenario 
Independent  Metrics  (SIM)  that  could  be  applied  to 
a  wide  range  of  future  exercises. 
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Metrics  Development 


•  Due  to  limited  training  resources  the  same 
emergency  scenario  is  almost  never  used  twice. 

•  SIM  offer  one  opportunity  to  increase  the 
continuity  between  emergency  management 
exercises  by  addressing  common  elements  of  a 
wide  range  of  emergency  response  scenarios 


What  you  lose  in  fidelity  you  gain  in  sample 
size  and  improved  continuity  between  exercises 
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Metrics  Development 


•  A  series  of  evaluation  tools  were  developed  to 
assess; 


-  Command  and  Control 

-  Communications 

-  Interdepartmental  Information  Sharing 


The  components  contained  in  each  of  the 
evaluation  tools  were  determined  through  review  of 
the  guidance  provided  by  the  U.S.  Homeland 
Security  Exercise  and  Evaluation  Program 
(HSEEP),  and  discussion  with  subject  matter 
experts  with  experience  in  each  respective  area  of 
the  evaluation 
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Scenario  Overview 


•  The  general  scenario  for  the  exercise  was  divided 
in  two  geographic  areas  with  three  main  events 

-  While  at  anchor,  a  string  of  explosions  onboard 
a  container  ship  occur.  A  container  containing 
Sodium  hydrosulphite  ignites  producing  a 
plume  of  toxic  smoke  that  is  carried  inland 

-  A  body  of  a  foreign  national  non-crewmember 
was  found  near  the  stricken  vessel  with  ties  to  a 
terrorist  organization 

-  A  security  incident  at  an  waterside  oil 
production  facility  occurs  that  may  be  linked  to 
the  explosion  on  the  container  ship 
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Results 


rV 


rd 


Number  of  Locations  =12 

Operations  Synchronization  Assessment  Tool  (OS AT) 

-  38  responses 

Communications  Assessment  Tool  (CAT) 

-  26  responses 

Coordination  Group  Assessment  (CGA) 

-  1 0  responses 


Post  Exercise  Player  Assessment  (PEP A) 
-  32  responses 
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Results 


rV 


Number  of  Locations  =12 

Operations  Synchronization  Assessment  Tool  (OS AT) 

-  38  responses 

Communications  Assessment  Tool  (CAT) 

-  26  responses 

Coordination  Group  Assessment  (CGA) 

-  10  responses 

Post  Exercise  Player  Assessment  (PEP A) 

-  32  responses 
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RL>  Coordination  Group  Assessment 

L^ 


Meeting 

Group: 

Location: 

Time: 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly  Agree 

Not  Applicable 

l 

2 

3 

4 

5 

n/a 

Question 

Description 

Response 

1 

Overall,  the  critical  information  required  to  coordinate  the  regional  federal  resources  and 
overall  response  activities  was  available? 

1 

2 

3 

4 

5 

n/a 

2 

Technical  and/or  science-based  advice  was  available  as  required? 

1 

2 

3 

4 

5 

n/a 

3 

Additional  community  members  were  engaged  as  appropriate  to  provide  assistance  and 
advice  on  the  situation? 

1 

2 

3 

4 

5 

n/a 

4 

The  optimum  group  of  agency  representatives  was  assembled  to  coordinate  the  overall 
response 

1 

2 

3 

4 

5 

n/a 

5 

The  format  of  the  meeting  (participants,  information  sources,  and  required  actions)  was 
well  defined  prior  to  the  event 

1 

2 

3 

4 

5 

n/a 

6 

Overall,  the  coordination  group  was  able  to  carry  out  their  primary  functional 
responsibilities? 

1 

2 

3 

4 

5 

n/a 

Comments: 
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m  Dashboard  Results  (Coordination  Group) 


CGA 

Description 

Average  Response  Value 

i 

Overall,  the  critical  information 
required  to  coordinate  the  regional 
federal  resources  and  emergency 
response  activities  was  available? 

3.1  (+-  0.7) 

2 

Technical  and/or  science-based 
advice  was  available  as  required? 

3.0  (+-  0.8) 

3 

Additional  community  members 
were  engaged  as  appropriate  to 
provide  assistance  and  advice  on 
the  situation? 

2.4  (+-  1.4) 

4 

The  optimum  group  of  agency 
representatives  was  assembled  to 
coordinate  the  emergency 
response? 

4.0  (+-  0.9) 

5 

The  format  of  the  meeting 
(participants,  information  sources, 
and  required  actions)  was  well 
defined  prior  to  the  event? 

4.0  (+-  0.9) 

6 

Overall,  the  coordination  group 
was  able  to  carry  out  their  primary 
functional  responsibilities 

4.0  (+-  0.7) 

Numeric  Value  1  2  3  4  5 


Response  Very  Low  Low  Neutral  High  Very  High 


Comments 


•  What  these  results  indicated  was  that  there  was 
quite  good  support  for  the  current  structure  and 
process  related  to  the  FCG,  but  that  the  group’s 
situational  awareness  and  access  to  the  right 
external  expertise  during  meetings  was  low. 

•  Improving  situational  awareness  and  access  to 
external  expertise  are  ideal  candidates  for 
technology  solutions. 

•  Engaging  additional  communities  members  for 
advice  during  the  coordination  group  meetings 
received  the  lowest  score  in  the  assessment.  This 
would  indicate  that  efforts  should  focus  on 
technology  solutions  that  would  increase  access  to 
relevant  community  members  as  required. 
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Lesson  Learned 


•  Try  to  establish  common  measurement  goals 
among  stakeholders  and  introduce  your  metrics 
early. 

•  Take  advantage  of  exercise  planning  conferences 

•  Accept  that  not  everyone  is  as  excited  about 
performance  measurement  as  you  might  be 


Try  to  build  good  relationships  with  the  agencies 
that  will  be  participating  in  the  evaluation 

Ensure  that  the  metrics  are  not  relegated  to  a  low 
priority  during  hectic  exercise  play 


Try  to  team  your  observer  with  a  SME  from  each 
respective  agency 


Defence  R&D  Canada  •  R  &  D  pour  la  defense  Canada 


f  X^y7  Future  Work 


•  Work  to  establish  self-reporting  standards  for 
agencies  participating  in  emergency  management 
exercises 

-  This  would  promote  more  effective  data 
management 

-  Reduce  the  number  of  external  evaluators 


Reduce  the  number  of  evaluators  required  at 
busy  operations  centers 
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Conclusions 


•  The  results  of  this  assessment  best  serve  as  a 
benchmark  for  comparison  with  future  evaluations 


•  As  new  operating  procedures  and  technology 
solutions  are  introduced,  we  can  now  begin  to 
explore  the  positive  or  negative  affect  these 
changes  have  with  respect  to  the  existing 
assessment  results  gathered  during  this  exercise. 


The  lessons  learned  from  this  exercise  will  help  to 
refine  the  concepts  and  questions  that  made  up  this 
evaluation 
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